Fix invalid values in cluster creation script #935
Conversation
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
parthosa
added
bug
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
| @@ -165,6 +165,9 @@ def get_matching_executor_instance(self, cores_per_executor): | |||
| def generate_cluster_configuration(self, render_args: dict): | |||
| executor_names = ','.join([f'"test-node-e{i}"' for i in range(render_args['NUM_EXECUTOR_NODES'])]) | |||
| render_args['EXECUTOR_NAMES'] = f'[{executor_names}]' | |||
| image_version = self.configs.get_value('clusterInference', 'defaultImage') | |||
| render_args['IMAGE'] = f'"{image_version}"' | |||
| render_args['ZONE'] = f'"{self.cli.get_zone()}"' | |||
There was a problem hiding this comment.
The zone should be picked up at the beginning of the execution.
I remember it was part of the getting the platform information during initialization.
My worry in this part that we are going to rely on the SDK CLI being installed which get us more restricted.
There was a problem hiding this comment.
Yes, we set env_vars dict during platform setup from the SDK config file. However, zone is not available in AWS' config file. Only region is available.
Now that makes me think, if we need zone in the cluster creation script for EMR? Why dont we use region instead.
There was a problem hiding this comment.
I guess it depends on what are the accepted keys under Ec2InstanceAttributes
If region is not one of the attributes, then we cannot use region there.
I don't remember exactly why did I have the availabilityZone part of the script. MAybe I tried to create a cluster and it required me to set the availability zone..I cannot really remember. It has been long time since I wrote that.
| @@ -165,6 +165,9 @@ def get_matching_executor_instance(self, cores_per_executor): | |||
| def generate_cluster_configuration(self, render_args: dict): | |||
| executor_names = ','.join([f'"test-node-e{i}"' for i in range(render_args['NUM_EXECUTOR_NODES'])]) | |||
| render_args['EXECUTOR_NAMES'] = f'[{executor_names}]' | |||
| image_version = self.configs.get_value('clusterInference', 'defaultImage') | |||
| render_args['IMAGE'] = f'"{image_version}"' | |||
| render_args['ZONE'] = f'"{self.cli.get_zone()}"' | |||
There was a problem hiding this comment.
I guess it depends on what are the accepted keys under Ec2InstanceAttributes
If region is not one of the attributes, then we cannot use region there.
I don't remember exactly why did I have the availabilityZone part of the script. MAybe I tried to create a cluster and it required me to set the availability zone..I cannot really remember. It has been long time since I wrote that.
Fixes #934. This PR fixes the invalid values of zone and image version for dataproc and emr cluster creation scripts.
Changes:
Image Version:
image versionin mustache template of dataproc and emr.image versionin config files.Zone:
{{ ZONE }}-afrom template file.aws cliand select the first one.Testing
CMD:
Manually tested on Dataproc and EMR.